Semi-Markov Models for Named Entity Recognition

We described semi-Markov models which relaxes usual Markov assumptions made in hidden Markov models. Semi-Markov models classify segments of adjacent words, rather than single words. We proposed two training strategies, a discriminative training and a generative training for semi-Markov models. Importantly, features for semi-Markov models can measure properties of segments, and transitions within a segment can be non-Markovian. This formalism can incorporate information about the similarity of extracted entities to entities in an external dictionary. This is difficult because most high-performance named entity recognition systems operate by sequentially classifying words as to whether or not they participate in an entity name; however, the most useful similarity measures score entire candidate names. In addition to allowing a natural way of coupling high-performance NER methods and high-performance similarity functions, this formalism also allows the direct use of other useful entity-level features, and provides a more natural formulation of the NER problem than sequential word classification. We applied semi-Markov models to named entity recognition (NER) problems and in experiments of ??????????????????????